I'm working on integrating Vapi CustomLLM with LangGraph, aiming to return streaming responses using fastify-sse-v2 for streaming delivery. Here’s a simplified illustration: // Run the first node invokeGreetUser() // Use a sleep function to check if the voice assistant begins speaking based on the first node’s response sleep(4000) // Run the second node invokeExplainInstruction(); // Terminate the stream reply.sseContext.source.end() Each node returns response chunks progressively. Here’s an example of a typical chunk: invokeGreetUser response data: {"id":"12345","choices":[{"delta":{"content":"","role":"assistant"},"finish_reason":null}]...} data: {"id":"12345","choices":[{"delta":{"content":"Hi Michael"},"finish_reason":null}]...} data: {"id":"12345","choices":[{"delta":{"content":". Nice to meet you"},"finish_reason":null}]...} invokeExplainInstruction response data: {"id":"12345","choices":[{"delta":{"content":"\nToday we are going to","role":"assistant"},"finish_reason":null}]...} data: {"id":"12345","choices":[{"delta":{"content":"talk about weather"},"finish_reason":null}]...} data: {"id":"12345","choices":[{"delta":{"content":". do you have any questions?"},"finish_reason":null}]...} // the response below is sent because it's the end of LangGraph data data: {"id":"12345","choices":[{"delta":{"content":null},"finish_reason":"stop"}]...} data: [DONE] The issue: Despite starting the streaming process, the voice assistant doesn’t begin speaking until the streaming ends. Could you provide any insights or suggestions on improving this flow for real-time speech initiation? Thanks you

Hey @erwin checking if this is resolved for you post office hours?

Hey @Shubham Bajaj , I just gave it a few tests, unfortunately It didn't work well.

hey @erwin can you share the call id for which flush is not working, meanwhile your reference sharing an example below.

json
data: {"id":"12345","choices":[{"delta":{"content":"Hi there <flush />"},"finish_reason":null}]...}

Hey @Shubham Bajaj, Here is the call id:

a4baf7c3-ead3-43a5-9b26-808110730f1a

. > Also do ensure chunkingPlan enabled is set to true else it won't work. I’ve already made sure it’s enabled on my end. Thanks for looking into this. https://cdn.discordapp.com/attachments/1304509416363659314/1305568891249889300/Screenshot_2024-11-10_at_07.44.10_1.png?ex=67338147&is=67322fc7&hm=326265733a415847baae7e8a83e2d0285331271e4e819fc3790e281f69a03643&

Have the same issue @Shubham Bajaj

logs

🔵 00:36:54:417 [user LOG] Model output: Thanks. I've noted that you were born in 1996. 🔵 00:36:54:417 PlayHT (Stream#1) Pushing "Thanks. I've noted that you were born in 1996. "... 🔵 00:36:54:417 Voice Input Formatted: "Thanks. I've noted that you were born in 1996. ", Original: "Thanks. I've noted that you were born in 1996. " 🔵 00:36:54:417 [user LOG] Voice input: Thanks. I've noted that you were born in 1996. Hey @erwin as you seen in the logs we are flushing out your model output, now to confirm if it's not working as expected can you share a public call recording URL or use Vapi storage for one of the call.

Hi @Shubham Bajaj, Sure, I'll share the record once I got the access permission from my team. By the way, I've created a simple code to re-create this issue. Repository https://github.com/erwin-huang/vapi-custom-llm Endpoint http://localhost:8000/api/v1/chat/completions Response (Static)

data: {"id":"msg_01KAzxVD1BoJDjsStAKHokJr","choices":[{"delta":{"content":"","refusal":null,"role":"assistant"},"index":0,"logprobs":null}],"created":1731491397,"model":"claude-3-5-haiku-latest","object":"chat.completion.chunk","service_tier":null}

data: {"id":"msg_01KAzxVD1BoJDjsStAKHokJr","choices":[{"delta":{"content":"Thanks. I've noted that you were born in 1996. <flush />","refusal":null},"index":0,"logprobs":null}],"created":1731491397,"model":"claude-3-5-haiku-latest","object":"chat.completion.chunk","service_tier":null}

data: {"id":"msg_01KAzxVD1BoJDjsStAKHokJr","choices":[{"delta":{"content":"Where did you grow up?","refusal":null},"index":0,"logprobs":null}],"created":1731491397,"model":"claude-3-5-haiku-latest","object":"chat.completion.chunk","service_tier":null}

data: {"id":"msg_01KAzxVD1BoJDjsStAKHokJr","choices":[{"delta":{"content":null,"refusal":null},"finish_reason":"stop","index":0,"logprobs":null}],"created":1731491397,"model":"claude-3-5-haiku-latest","object":"chat.completion.chunk","service_tier":null}

data: [DONE]

Thanks

Hey @erwin just checking if this is resolved for you, as I got to know Sahil has helped you resolving this?

Hey @Shubham Bajaj, Finally it worked on my end, but it stops when there is a dot (

"."

) character.

data: {"id":"12345","choices":[{"delta":{"content":"Hi Michael. My name is erwin <flush />"},"finish_reason":null}]...}

 data: {"id":"12345","choices":[{"delta":{"content":" "},"finish_reason":null}]...}

The voice assistant will only say "Hi Michael.". Is there a way to overcome this? And also I noticed when I use more than one

<flush />

in the stream, only the first one will work.

data: {..."choices":[{"delta":{"content":"Hi Michael this is a description <flush />"},"finish_reason":null}]...}

 data: {..."choices":[{"delta":{"content":" "},"finish_reason":null}]...}

 data: {..."choices":[{"delta":{"content":"This is another description <flush />"},"finish_reason":null}]...}

 data: {..."choices":[{"delta":{"content":" "},"finish_reason":null}]...}

The voice assistant will only say "Hi Michael this is a description". Is there anything I can do to make the

<flush />

work on the second sentence? Thanks for helping out

Hey @erwin That's great to hear it's working for you now! Just so you know - O get to know yesterday that there's a small issue with the

<flush />

feature. We're working on fixing it right now. I'll make sure to let you know as soon as it's all sorted out!

Sounds good, thanks for the heads up Looking forward to the fixes @Shubham Bajaj